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® Signal identification system. 



© A system for identifying signals such as signals 
representative of television programs received from 
various sources detects the occurrence of predeter- 
mined events such as scene changes in the video 
signal and extracts a signature from the signal. The 
signatures and the times of occurrence of the signa- 
ture are stored and subsequently compared with 
reference signatures to identify the signal. The sig- 
nature may be compared in pairs to increase resolu- 
tion, and the time interval between events or signa- 
tures may also be used to identify the signal, either 
j by themselves or in conjunction with the signatures. 
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Signal identification system 



This invention relates to a system for identify- 
ing signals such as signals representative of televi- 
sion programs which may be obtained from various 
sources including video recorders. 

Systems for identifying programs such as tele- 
vision programs that are broadcast or viewed are 
known. Such sytems fall into various categories. 
The categories include manual systems wherein 
the viewer keeps a diary indicating all of the pro- 
grams viewed; mechanical, electro-mechanical and 
electronic systems that sense the channel to which 
a television receiver is tuned; systems that detect 
identifying signals present in the television broad- 
cast signal; and systems that monitor and analyze 
the program content of the broadcast signal utiliz- 
ing image processing and correlation techniques to 
identify the program. 

While all of these systems do provide a way to 
identify programs and to monitor the viewing habits 
of the public, the manual methods are slow in 
acquiring data and are prone to inaccuracies result- 
ing from the entry of erroneous data that may be 
intentionally or unintentionally entered. Systems 
that monitor the channel to which a receiver is 
tuned overcome the drawbacks of the manual sys- 
tem, but require access to the tuning mechanism of 
the receiver. Such access is becoming increasingly 
difficult with the advent of cable and pay television 
systems that utilize various converters and decod- 
ers, many of which are difficult to access. More- 
over, such systems cannot identify programs 
played from a source such as a home video re- 
corder. Systems that detect identification signals 
encoded on the broadcast signal overcome many 
of the accuracy and access disadvantages de- 
scribed above, but require the cooperation of the 
broadcasters who must encode the programs prior 
to broadcast for such systems to be effective. 
Consequently, such systems are useable only on a 
limited number of broadcasts, such as network 
broadcasts. Systems that monitor and analyze the 
content of the program itself theoretically eliminate 
the disadvantages of the other systems, but here- 
tofore the information processing requirements of 
such systems rendered such systems impractical 
or uneconomical because of the computing capac- 
ity required to analyze the vast quantity of informa- 
tion present in broadcast signals. Moreover, pre- 
vious attempts to reduce the quantity of information 
processed tended to introduce inaccuracies. 

An example of a system that utilizes the pro- 
gram content of a broadcast signal to identify the 
program is described in United States Patent No. 
3,919,479 to Moon et ai. the Moon et al. system 
utilizes a non-linear analog transform to produce a 



low frequency envelope waveform, and the infor- 
mation in the low frequency envelope of a pre- 
determined time interval is digitized to generate a 
signature. The signatures thus generated are com- 
5 pared with reference signatures to identify the pro- 
gram. However, the Moon et al. system generates 
the signatures continuously, and consequently, re- 
quires a large computer to process the data thus 
generated. 

10 Another system that monitors the program con- . 

tent of a broadcast signal is disclosed in United 
States Patent No. 4,230,990 to Lert Jr. et al. The 
system disclosed in the Lert Jr. et al patent re- 
duces the amount of data that must be processed, 

75 as compared to the data amount as processed by 
the Moon et al. patent, by utilizing cues that are 
either externally generated or present in the pro- 
gram material ti initiate the signature generation 
and correlation process only after a cue has been 

20 detected. However, the system disclosed in the 
Lert Jr. et al. patent is designed to monitor the 
programs broadcast by broadcasting stations such 
as network affiliated stations in order to determine 
whether they are broadcasting the programs re- 

25 quired by the networks to be broadcast, rather than 
as a system for determining the viewing habits of 
the public. Thus, many of the problems that occur 
when the habits of a television viewer are being 
monitored, such as, for example, frequent channel 

30 changing, and the problems that occur in identify- 
ing programs that are recorded and subsequently 
played back by a home video recorder are not 
addressed. 

The invention as claimed solves the problem of 

35 identifying signals such as signals representing 
programs broadcast and the viewing habits of tele- 
vision viewers regardless of the source of the pro- 
gram material, whether it be off the air, cable or 
prerecorded material, utilizing data reduction and 

40 correlation techniques to minimize the amount of 
computation required to identify the signal. This is 
accomplished by extracting signatures from the 
signal only upon the occurrence of certain pre- 
determined events, whether the events occur e.g. 

45 in the program material, for example a blank frame 
or a scene change, or are the result of an action 
taken by the viewer, for example the on/off con- 
dition of the television set and the mode of opera- 
tion of the video recorder. In addition to providing a 

so trigger for the extraction of the signature from the 
signal, the predetermined sequence of events may 
serve to indicate whether a program is being re- 
corded or a prerecorded program is being viewed. 

In accordance with a preferred embodiment of 
the invention, there is provided a system having a 
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home unit that monitors the viewing conditions in 
the home, for example, whether the television set is 
on or off, whether any home video recorder (e.g., a 
video cassette recorder or VCR) is on or off and 
the video signal being received. By monitoring the 
aforementioned signals and functions, the system 
determines whether no viewing is occurring, wheth- 
er a television broadcast is being viewed, whether 
a television broadcast is being recorded, whether 
viewing and recording is occurring simultaneously, 
or whether a previously recorded program is being 
viewed. Once the viewing mode has been estab- 
lished, the program being viewed is identified by 
extracting a characteristic signature from the video 
signal and storing it for subsequent comparison 
with reference signatures stored in a central office. 

Once the mode of viewing has been deter- 
mined, a signature of the program material being 
viewed or recorded is extracted, provided certain 
predetermined events occur first. These events 
shall be referred to as Event 1 and Event 2. Sev- 
eral events can serve as Event 1 . These events 
may be a turn on of the television set, a channel 
change, a predetermined time interval, for example 
5 to 10 minutes, since the occurrence of the pre- 
vious Event 1, or a sudden change to a black 
scene. Events that can serve as an Event 2 include 
a scene change to a scene other than a black 
scene, a color change, which may be an overall 
color change or a color change of a predetermined 
line, a still picture as evidenced by several con- 
secutive substantially identical frames, an audio 
level change, and others. 

Upon the occurrence of an Event 1 , the home 
unit monitors the broadcast for the occurrence of 
an Event 2, and extracts a pair of signatures from 
the video signal immediately following the next two 
Event 2s following an Event 1 . Or, if a previously 
recorded program is being viewed, the system 
extracts three or more signatures from the video 
signal immediately following the next three or more 
Event 2s following an -Event 1 . The signatures are 
extracted be detecting and digitizing the envelope 
of the video signal immediately following the Event 
2s. The digitized signatures, as well as the times 
that the signatures were extracted, are stored for 
future comparison with reference signals stored in 
a central location. The time intervals between Event 
2s are also stored and used to identify the program 
either by themselves or in conjunction with ex- 
tracted signatures. The time intervals between 
Event 2s are particularly useful for identifying the 
playback of previously recorded video tapes for 
reasons that will be discussed in a subsequent 
portion of the specification. 

The unit used to acquire the reference signa- 
tures that are compard with the signatures received 
from the home units is similar to the home units 



except that the reference signature acquisition unit 
monitors each broadcast continuously, and does 
not require an Event 1 to initiate a signature extrac- 
tion. Therefore, each broadcast is continuously 

5 monitored and a signature is extracted each time 
an Event 2 is detected. The signatures, as well as 
the times of occurrence of the signatures are 
stored to form a library of reference signatures. 
The stored signatures are compared with signa- 

w tures obtained from the home unit that occurred at 
approximately the same time. The time intervals 
between Event 2s are also stored for comparison 
with corresponding time intervals received from the 
home units. 

75 A mode of implementation and the major ad- 

vantages of the present invention will become read- 
ily apparent upon consideration of the following 
detailed description and attached drawing wherein: 
FIG. 1 is a block diagram of a home unit of a 
20 program identification system according to the 
present invention; 

FIG. 2 is a block diagram of a central office 
and reference signature extraction portions of a 
system according to the invention; 
25 FIG. 3 is a chart illustrating the various 

modes of viewing and recording that can occur in a 
typical household; 

FIGS. 4 and 5 are flow charts illustrating the 
logical steps performed by the home and central 
30 office units, respectively; 

FIG. 6 is a flow chart illustrating the logical 
steps performed by the central office unit in iden- 
tifying signatures; and 

FIG. 7 illustrates how signatures and scene 
35 changes may be extracted from the video en- 
velope. 

Referring now to the drawing, with particular 
attention to FIG. 1, there is illustrated a block 
diagram of a home unit of the system according to 

40 the invention generally designated by the reference 
numeral 10. The system 10 receives signals to be 
identified from a television reciever 12, and in 
many instances from a home video recorder, such 
as a' video cassette recorder or a VCR 14. In a 

46 typical home installation, the receiver 12 and the 
VCR 14 receive signals from a receiving antenna 
16, and in many instances from a cable television 
system 18. These signals may be applied directly 
to the receiver 12 and to the VCR 14, but it is 

so convenient to employ a switching network 20 that 
selectively switches the receiver 12 and VCR 14 to 
the antenna 16 and to the cable 18 in order to 
permit the viewing and recording of broadcasts 
received from the antenna 16 or from the cable 18. 

55 In addition, the switching network 20 may connect 
the VCR 14 to the receiver 12 to permit prerecor- 
ded or previously recorded tapes to be played 
back by the VCR and viewed by the receiver 12. 
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Video signals representative of the program 
applied to the receiver 12, as well as video signals 
representative of the signals being recorded by the 
VCR 14, are applied to a video processing circuit 
22, for example, by a video switching circuit 24 
which permits either the receiver 12 or the VCR 14, 
or both, to be monitored. The video processing 
circuit 22 includes a detector and a low pass filter 
(discussed subsequently in greater detail), and pro- 
vides a signal representative of the envelope of the 
video signal to an analog-to-digital converter 25 
which generates a digital representation of the vid- 
eo envelope. The digitized video envelope signal is 
applied to an events detector 26 which detects 
predetermined events that occur in the digitized 
video signal, and causes the signature of the 
digitized video signal to be extracted by signature 
extraction circuitry 28 upon the occurrence of a 
predetermined sequence of events. The signature 
thus generated is stored in a data storage system 
30 along with the time that the signature was 
extracted. The extraction times are provided by a 
clock 32 which provides a digital representation of 
time to the data storage system 30 via, for exam- 
ple, a combining circuit 34. The clock 32 may be a 
real time clock or a relative time clock that is 
periodically reset from a central location. The func- 
tion of the events detector 26, the signature extrac- 
tor circuitry 28, the data storage 30 and the clock 
32 can be performed by individual circuits as 
shown or by a microprocessor based system. 

The home unit 10 is interrogated at periodic 
intervals, for example, once a day, by a central 
office unit, generally designated by the reference 
numeral 40 (FIG. 2), which compares the signa- 
tures from the various home units with reference 
signatures in order to identify the signatures from 
the home unit. The home units may be periodically 
interrogated by a data communications circuit 42 
that accesses the various home units via a suitable 
communications system, for example, a plurality of 
telephone lines 44. The signatures thus collected 
are compared by a central computer system 48 
with signatures stored in a data base 46. Central 
computer system 48 controls the collection and 
classification of the signatures received from the 
home unit as well as the generation of reference 
signatures to be stored in the data base 46. The 
latter function is performed in conjuction with a 
plurality of reference signal extraction circuits 50 
that are located in the cities being monitored. The 
reference signature extraction circuits monitor the 
various networks, cables and other signal sources 
in those cities, and extract reference signals, for 
example, whenever an Event 2 occurs in any of the 
signals being monitored. The reference signals 
thus extracted are transmitted to the central office 
unit 40 and stored in the data base 46 along with 



the times that such signatures are extracted, as 
provided by a clock 52. The clock 52 of FIG. 2 is 
similar to the clock 32 of the home unit and serves 
to indicate the time of occurrence of the reference 

5 signatures. The clock 52 may also be a real time 
clock or a relative time clock that is periodically set 
by the central office 40. A central reference signa- 
ture extraction circuit 54 and associated clock 56 
serve to extract signatures and times of extraction 

10 of cable originated programs and signatures of 
prerecorded programs. Alternatively, extraction cir- 
cuits may be placed at the head ends of cable 
system to extract reference signatures of cable 
programs. The signature extraction performed by 

is the various extraction circuits and data storage is 
controlled by the central computer system 48. 

The home unit of the system monitors the 
mode of viewing by monitoring the on and off and 
other functions of the television receiver and any 

20 video recorder that may be used in the home. The 
various modes of viewing or recording are illus- 
trated in FIG. 3. Mode 1 occurs when both the VCR 
and television receiver are both off, and indicates 
that no viewing is taking place. Mode 2 represents 

25 television viewing and occurs when the television 
receiver is on and the VCR is off. Mode 3 repre- 
sents recording by the home VCR and occurs 
when the television reciever is off and the VCR is 
on. Mode 4 occurs when the VCR and the televi- 

30 sion receiver are both on, and can represent one of 
three conditions. The first condition, designated as 
mode 4.0, represents television viewing through the 
tuner of the VCR without recording. Mode 4.1 re- 
presents VCR recording and television viewing, 

35 while the third condition, designated as Mode 4.2, 
represents VCR playback. Other modes of opera- 
tion of the VCR, such as fast forward and rewind, 
are designated as Modes 4.3 and 4.4, respectively. 
Most of these modes can be easily detected by 

40 monitoring power line voltage or voltages else- 
where in the television set and the video recorder. 
Modes 4.0, 4.3 and 4.4 can be detected by logic 
circuitry. 

As previously stated, the home unit 10 moni- 
es tors the viewing mode and the video signal, and 
extracts signatures from the video signal only if 
certain events occur. The logic necessary to pro- 
vide this function is illustrated in FIG. 4. As is 
apparent from FIG. 4, the mode of operation is 
so recorded whenever there is a change in the mode, 
such as, for example, a change from viewing to 
recording, etc. Each time the mode changes, a 
determination is made in order to determine what 
the new mode is. This is accomplished by deter- 
55 mining whether the television is on or off, whether 
the VCR is on or off, and whether the VCR is in a 
record or playback mode. Once it has been deter- 
mined whether the television and VCR are on or 
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off, and whether the VCR is in a record or playback 
mode, the mode can readily be determined by 
using a look-up table containing the information in 
FIG. 3, and the mode is recorded. 

If the mode is determined to be Mode 1, in- 
dicating no viewing, no further action is taken until 
the mode changes again. If it is determined that 
the mode of operation is either Mode 2 or Mode 4, 
the system monitors the television reciever for the 
occurrence of an Event 1, which may, for example, 
be any one of the following: 

1 . television turn-on 

2. a channel change 

3. a predetermined time interval (e.g 5-10 
minutes) since the previous Event 1 

4. a sudden scene change to a black scene. 

5. loss of synchronization for an appreciable 
period of time. Such Event 1s are relatively easy to 
detect. For example, television turn on can be 
detected simply by monitoring an appropriate volt- 
age or current. Televi sion turn-on is not used as 
an Event 1 in the system illustrated in FIG. 4 but 
may be used in other sytems, particularly systems 
that do not monitor a VCR and consequently need 
not employ mode logic. A channel change can be 
detected by monitoring the position of the tuner 
mechanism when mechanical tuners are used, by 
monitoring tuning voltage in electrically tuned tun- 
ers, by monitoring iocal oscillator frequency, or 
even more simply by monitoring the video syn- 
chronization pulses, preferably the vertical syn- 
cronization pulses and indicating a .channel change 
in the event of a loss or a change in the syn- 
cronization pulse for a short time interval. The 
predetermined time interval can readily be deter- 
mined from a clock, and sudden scene change to 
black can be readily detected by monitoring the 
average value of the video signal and indicating a 
change to a black scene when the average value 
becomes that representative of an all black scene. 
A loss of synchronization for an appreciable length 
of time (longer than that caused by a channel 
change) indicates fast forward or rewind mode of 
operation of a VCR (Mode 4.3 or 4.4) or the loss of 
the video signal (e.g. station off the air). 

As long as no Event 1 occurs, the home unit 
records no new data; however, upon the occur- 
rence of an Event 1 , the system is conditioned to 
extract a signature upon the occurrence of the next 
two consecutive Event 2s, with an Event 2 being 
defined as: 

1 . a scene change to a scene other than a 
black scene 

2. a color change in successive frames or 
portions thereof 

3. a still picture 



4. an audio level change. Ways of detecting 
the various Event 2s described above, as well as 
ways for extracting signatures will be discussed in 
a subsequent portion of the specification. 

5 Upon the occurrence of an Event 1 , a counter 

is set to zero and the video signal is monitored to 
determine if the television synchronization or sync 
signal is stable. This is accomplished, for example, 
by monitoring the vertical synchronization pulses 

io and waiting until a predetermined number, for ex- 
ample, 16, consecutive stable vertical sync pulses 
are detected before the unit is permitted to detect 
an Event 2. If the synchronization has not been 
stable for a significant period of time, more than 

is five seconds, this period will be marked either as 
television station is off the air or VCR fast forward 
or rewind- 
Once the sync has been stable for the required 
number of pulses, the system monitors the video 

20 signal for the occurrence of an Event 2. Whenever 
such an Event 2 change is detected, the time of 
the occurrence of the Event 2 is recorded as is the 
time interval since the previous Event 2, if the time 
intervals between Event 2s are to be used as a 

25 means of program indentification. The signature of 
the first frame following the last detected Event 2 
change is then extracted. Since a standard broad- 
cast telelvision fram consists of two interlaced 
fields, and since the two fields that form a frame 

30 are quite similar, it is not necessary to analyze both 
fields of the frame to obtain a signature. Typically, 
a satisfactory signature is obtained from only a 
single field, for example, the first field of a frame. 
The method of extracting the signature will be 

35 discussed in greater detail in a subsequent portion 
of the detailed description, as will the method of 
detecting an Event 2. 

Once the signature has been extracted, the 
counter is incremented by 1. If the count in the 

40 counter is less than two, indicating that more signa- 
tures are needed, the system is conditioned to 
respond to another Event 2 to extract and record 
another signature. If the count in the counter is two 
(or the required number of signatures) or greater, 

45 indicating that the signatures required have been 
extracted, the system is conditioned to require an 
Event 1 to occur before it responds to any further 
Event 2s. 

If it is determined that the mode of operation is 
so Mode 3, that is, VCR recording, it is not necessary 
for an Event 1 to occur before the signal is mon- 
itored for Event 2s. The only determination that is 
made is whether the synchronzation is stable, and 
if so, the system looks for Event 2s continuously 
55 and extracts the signature and records the time of 
occurrence of the signature each time an Event 2 
is detected. Thus, ail signatures following every 
Event 2 are extracted and stored so that upon 
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subsequent playback, the program may be iden- 
tified even though only portions ofthe tape may be 
played back. 

The logic for extracting reference signatures is 
illustrated in FIG. 5. Each broadcast and cable 
channel is monitored continuously by the reference 
signature extraction circuits 50 and 54, and a sig- 
nature is extracted each time an Event 2 occurs to 
provide a reference library of reference signals. 
Consequently, since it is desired to extract a signa- 
ture that occurs after every Event 2, Event 1s are 
not monitored. All that is required is that the sync 
be stable and that an Event 2 occur. Thus, when- 
ever an Event 2 occurs, the time of occurrence of 
the Event 2 as well as the signature of the scene 
following the scene change is recorded for future 
comparison with signatures received from the var- 
ious home units. The time intervals between Event 
2s may be recorded or obtained from the times of 
occurrence of the Event 2s. 

The logic employed by the central computer 
system 48 of the central unit 40 in order to identify 
a program is illustrated in FIG. 6. As is illustrated in 
FIG. 6, two consecutive signatures from a home 
unit are selected. If there is not enough data to 
define two consecutive signatures, the next home 
unit is monitored. After two consecutive signatures 
are selected, a check is made to determine wheth- 
er the signatures really are a pair, for example, that 
there is no Event 1 between the two signatures. 

If the two signatures are a pair, the time asso- 
ciated with the first home unit signature is read.. All 
reference unit signatures in the data base 46 that 
occurred within a predetermined time interval of 
the occurrence of the first home unit signature, for 
example, plus or minus eight seconds, are selected 
for comparison, and the correlation coefficients be- 
tween the first home unit signature and the se- 
lected reference unit signatures are computed. If 
one of the correlation coefficients exceeds a pre- 
determined threshhold, the time associated with the 
second home unit signature is read. The next sev- 
eral, for example, six, reference unit signatures that 
occur, for example, within a predetermined time 
interval, for example, plus or minus eight seconds 
of the time associated with the second home unit 
signature are selected, and the correlation coeffi- 
cients between the second home unit signature and 
the reference unit signatures are computed. If one 
of the correlation coefficients exceeds the predeter- 
mined threshhold, an identifica tion is made and 
stored, and printed out, if necessary. 

As discussed above, the system according to 
the invention utilizes a time reference to access 
reference signatures that occurred at approximately 
the same time as the home unit signature being 
evaluated in order to eliminate the need for com- 
paring the home unit signature with all of the refer- 



ence signatures in the data base 46. However, 
when a previously recorded program is being 
played back, the time that the signature occurs 
during playback cannot be used to iocate the refer- 

5 ence signal. Thus, in accordance with another im- 
portant aspect of the present invention, the times of 
occurrence of signatures that occur when programs 
are being recorded are also stored. The times of 
occurrence of signatures that occur in Mode 3 

w (recording) are kept in a separate recording time 
file in the data base 46. Similarly, in Mode 4,1 
(viewing and recording) the times of signatures are 
also kept in the recording time file. Thus, when the 
broadcast is played back (Mode 4.2), even though 

15 the recorded signatures cannot be matched to sig- 
natures stored in the reference unit during 
playback, the signatures can still be identified by 
utilizing the recording time instead of the playback 
time to retrieve the corresponding reference signa- 

20 tures from the reference signature data base 46. 
Thus, by keeping signatures relating to programs 
that were broadcast several days or weeks ago 
within the data base 46, such previously recorded 
and subsequently played back broadcasts can be 

25 readily identified. 

Alternatively, in order to identify the playback 
of previously recorded programs, instead of com- 
paring those obtained during playback with refer- 
ence signatures that represent the programs broad- 

30 cast several days or weeks ago stored in the 
recording time file of the reference signature data 
base 46 (FIG. 2) of the central office unit, the 
played back signatures may be compared with 
signatures obtained by the home unit previously 

35 during VCR recording (Mode 3) and stored in the 
data storage 30 (FIG. 1) of the home unit. If de- 
sired, the signatures may also be sent to the cen- 
tral office, as are the signatures obtained during 
normal television viewing, but these signatures 

40 would be labeled as VCR recording, and may be 
stored in the recording time file. However, when 
playback occurs, the played back signatures would 
be compared with the signatures stored in the 
home unit data storage. Only if a match occurred, 

45 would the signatures be sent to the central office 
and compared with the signatures stored in the 
central office data base 46 to identify the program. 
If no match occurred, the signatures would be sent. 
This is because whatever is being played back 

so cannot be identified, and may be something like, 
for example, a home movie. Thus, the system 
eliminates the need to search the entire recording 
file of the data base 46 in an attempt to identify 
something that cannot be identified. 

55 In the playback mode, when attempting to 
mate the signatures obtained during playback with 
the signatures stored in the data storage 30, it is 
convenient to use the time intervals between the 
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occurrence of the signatures (or Event 2s) to gen- 
erate a time interval signature, TIS, (discussed in 
greater detail in a subsequent portion of the speci- 
fication) to obtain the match. When using a time 
interval signature for matching, three or more con- 
secutive signatures are extracted, and the time 
intervals between the signatures are compared with 
the time intervals between signatures stored in the 
data storage 30. When a corresponding pair of 
consecutive time intervals is found, a match is 
indicated. If greater accuracy is desired, more than 
three consecutive signatures may be extracted to 
provide more than two consecutive time intervals. 

Event 2s may be detected in a variety of ways. 
For example, the video signal may be low pass 
filtered, for example, through a low pass filter hav- 
ing a cut-off frequency on the order of approxi- 
mately 2 to 6 kiloHertz, and its average amplitude 
detected. The signal may be digitized by assigning 
a number representative of the average amplitude 
of the video signal for each frame. The numbers 
thus obtained may be compared in various ways. 
For example, the numbers obtained from different 
frames may be compared, and if the numbers are 
substantially different a scene change will be in- 
dicated. The numbers compared may be obtained 
from consecutive frames or from non-consecutive 
frames, for example, frames spaced by a 20 to 40 
frame interval, depending on the type of event that 
has been defined as the Event 2. Alternatively, a 
moving average of the numbers representative of 
the video amplitudes of the fast several frames 
may be calculated, and the number representative 
of the average amplitude of the last detected frame 
compared with the moving average to indicate a 
scene change if the number associated with the 
last frame substantially differs from the moving 
average. 

The various comparing schemes are useful for 
identifying various different types of Event 2s. For 
example, comparing the number associated with a 
particular frame with the number associated with a 
previous frame is useful for detecting an Event 2 
that may be termed an instant scene change. A 
moving average comparison or a comparison with 
frames that occurred several frames ago is useful 
for detecting an Event 2 that may be termed a fade 
type scene change wherein one scene is gradually 
faded in and the previous scene is gradually faded 
out. Because in a fade type scene change, the 
change occurs gradually, there is no drastic 
change in the number associated with consecutive 
frames, and consequently, such a scene change 
cannot readily be detected by a system that com- 
pares immediately successive frames. Numbers 
that are substantially identical for several consecu- 
tive frames may be used to indicate a still picture. 

Other methods of detecting an Event 2 include 



detecting a color change, for example, by detecting 
the average amplitude of a particular color compo- 
nent, such as, for example, a red component, of 
each frame or a particular line or lines of each 

5 frame, and making comparisons of the types de- 
scribed above. Alternatively, different color compo- 
nents or an average of the three color signals can 
be monitored for each frame and comparisons 
made. In addition, rather than monitoring an entire 

io frame, only portions of a frame may be monitored. 
These portions may be selected to be those por- 
tions of a frame normally do not change unless a 
scene change occurs. Such portions may occur 
near the top or bottom of the screen because such 

75 areas are less iikely to be affected by action or 
movement occurring within a scene. The Event 2s 
may be detected electronically by monitoring the 
video envelope, or by monitoring the cathode ray 
tube, for example, by placing a light sensitive pho- 

20 toelectric device 36 (FIG. 1) in proximity with the 
cathode ray tube, preferably in one comer thereof. 
Alternatively, the audio signal may be monitored by 
an audio processing circuit 38 and applied to the 
analog-to-digital converter 25 to generate an Event 

25 2 when the audio signal departs from a predeter- 
mined level. 

In accordance with an alternative embodiment 
of the present invention that is particularly suitable 
for identifying prerecorded programs, the time in- 

30 tervals between events, or of changes in one or 
more predetermined parameters, can be used to 
define a feature string. Such a system can be used 
either alone, or in combination with a signature 
extracting system to identify the program. When 

35 used alone, the amount of data generated by the 
system is less than that generated by a signature 
generating system, and therefore, does not require 
much computation time to obtain a correlation. 
Consequently, it is not necessary to store the time 

40 of occurrence of each change (unless it is desired 
for other reasons), but rather, only the elapsed time 
between such changes need be stored. Such a 
system is particularly useful in identifying the 
playback of prerecorded programs, or delayed 

45 broadcast programs because real time data is not 
necessary to make the identification. 

In a system wherein the intervals between pre- 
determined changes in predetermined events are 
used to generate a time interval signature (TIS). 

so The time interval signature, which can be repre- 
sented as a digital code sequence, can be con- 
structed by measuring the time intervals between 
the sequential occurrences of regularly occurring 
events in a broadcast. Preferably, the chosen 

55 events should have the following properties; 

1. Be readily measurable automatically. 
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2. Occur at a sufficiently high rate that 
enough intervals are available to allow unique iden- 
tification of the signal. 

3. Always be present 

4. Cannot be controlled by program origina- 
tors to render identification vulnerable to changes 
in programming methods. 

Typical events that can be used as the events 
to generate a time interval string includes scene 
changes to black. Event 2s, including shifts in the 
average amplitude of the video signal, departures 
in the amplitude of the instantaneous video signal 
from the average, shifts in the video color balance, 
changes in the average color of the predetermined 
line, for example, a line near the top of the screen, 
a predetermined audio power level, for example, 
silence, etc. Thus, to define a time interval string, 
the home unit would generate a time interval string 
(TIS) by detecting one of the predetermined 
events, for example, an Event 2 such as a scene 
change, and store the time intervals between con- 
sectutive changes. 

The monitoring of events can be done continu- 
ously to generate a long TIS without requiring an 
excessive memory capacity because such changes 
occur relatively infrequently, and the time intervals 
therebetween can be defined by only a few bits. 
The TIS's thus obtained from the home unit are 
compared with reference TIS's stored in the central 
unit 

However, in order further to reduce the amount 
of data that must be stored, the time intervals 
between the events being monitored may be mon- 
itored only after the occurrence of another pre- 
determined event, such as an Event 1; and it has 
been found that satisfactory identification can be 
obtained by generating a feature string consisting 
of three time intervals between changes in the 
monitored variable, such as a scene change or 
other Event 2. The time interval strings thus gen- 
erated are matched with reference time interval 
strings that are obtained in much the same way as 
those generated by the home units except that the 
reference time interval strings are continuously 
generated for each broadcast or other program 
source being monitored. 

Referring now to Fig. 7, there is shown a video 
envelope 100 as provided from the video process- 
ing circuit 22 (Fig. 1) to the analog-to-digital con- 
verter 25. The analog-to-digital converter 25 takes 
a plurality of samples 102 (Fig. 7) for example, 80 
samples during a single frame interval, and gen- 
erates a digital value representative of the am- 
plitude of each of the samples 102. The digital 
.values thus generated are applied to the events 
detector 26 (Fig. 1) and to the signature extraction 
circuit 28 and used to detect the various events 
and to extract signatures. 



In order to detect an Event 2, for example, a 
scene change, the events detector 26 compares 
each of the digitized samples 102 of a current 
frame with a corresponding digitized sample 102 of 

5 a previous frame, either the immediately preceding 
frame or another previous frame, depending on the 
type of scene change to be detected. The coun- 
terpart samples are then compared to determine 
whether a scene change has occurred. More spe- 

io cifically, the detection of a scene change occurs as 
follows: 

1. Take, for example, 80 samples of the 
video envelope of each frame 

2. Compare counterpart samples of a current 
75 frame and a previous frame 

3. Take the absolute value of the difference 
of each of the counterpart samples 

4. Take the average of the absolute values 

5. Indicate a scene change if the average of 
20 the absolute values exceeds a predetermined 

threshhold. The indication of a scene change is 
applied to the signature extraction circuit 28 (Fig. 

1) to cause the signature extraction circuit 28 to 
operate on the digitized samples 102 to extract the 

25 signature. 

The signature extraction circuit 28 operates on 
the digitized samples 102 by taking the average 
value of consecutive samples 102, for example, 
eight consecutive samples to generate a total of, 

30 for example, ten features 104 that comprise an 
. image signature. Typically, four bits define a fea- 
ture, and consequently, an image signature can be 
defined by forty bits of data for the example dis- 
cussed above; however, in some instances more 

35 features and more data may be required. 

Both the home unit 10 (Fig. 1) and the re- 
ference signature extraction circuits 50 and 54 
(Fig. 2) operate in the same manner to extract 
signatures, the only difference in operation being 

40 that the signature extraction circuit 28 (Fig. 1 ) re- 
quires an Event 2 to be preceded by an Event 1 
before signature extraction occurs; but the refer- 
ence signature extraction circuits 50 and 54 (Fig. 

2) do not require an Event 1 to precede an Event 
45 2, but rather extract a reference signature each 

time an Event 2 occurs. 

Once the home unit signatures and the refer- 
ence signatures are obtained, the home unit signa- 
tures will be compared with the reference unit 

so signatures, normally with those signatures occur- 
ring in corresponding time intervals. The correlation 
coefficients between the home unit signatures and 
the reference unit signatures will be compared, and 
the reference signature that has the best correlation 

55 to the home unit signature will be used to identify 
the program being viewed, provided that the cor- 
relation coefficient exceeds a minimum threshhold. 
If it does not, no match will be indicated. 



8 



15 



0 283 570 



16 



Claims 

1. A system for identifying signals, the system 
having means (26, 28, 32, 34, 30) for operating on 
the signal to be identified in order to extract a 
feature string, and means (48) for comparing the 
feature string with various feature strings corre- 
sponding to known signals and for identifying the 
signal when the feature string of the signal to be 
indentified correlates with one of the feature strings 
corresponding to a known signal within a predeter- 
mined tolerance, wherein the feature string extac- 
ting means is characterized by - means (26) for 
monitoring predetermined changes in predeter- 
mined parameter of the signal to be identified, 

- means (26, 32) for determining the time intervals 
between the predetermined changes in the pre- 
determined parameter, and 

-means (28, 32, 34, 30) for utilizing the time inter- 
vals between successive predetermined changes to 
generate the feature string. 

2. The system recited in claim 1 wherein the 
signal to be identified includes an audio signal, and 
wherein said feature string extracting means (26, 
28, 32, 34, 30) inlcudes means (38, 26, 32) for 
monitoring the amplitude of the audio signal and 
detrmining the time intervals that the amplitude of 
the audio signal departs from a predetermined lev- 
el of amplitude. 

3. The system recited in claim 2 wherein the 
predetermined level amplitude is a level that cor- 
responds substantially to silence, and said time 
interval corresponds to the time interval between 
successive periods of silence in the audio signal. 

4. The system recited in claim 1 wherein the 
signal to be identified includes a video signal and 
wherein said feature string extracting means (26, 
28, 32, 34, 30) includes means (26, 32) for deter- 
mining events (Event 2) in the video signal and 
determining the time intervals between such events 
to generate the feature string. 

5. The system recited in claim 4 wherein said 
event determining means (26, 32) includes means 
for monitoring the light emanating from the display 
on which the video signal is displayed and indicat- 
ing a scene change when the amount of light 
emanating from the display changes by a predeter- 
mined amount. 

6. The system recited in claim 5 wherein the 
display light monitoring means includes means for 
monitoring the light with a photoelectric device 
(36), and indicating a respective event (Event 2) 
when the output of the photoelectric device 
changes by a predetermined amount. 

7. The system recited in claim 4 wherein the 
event determining means (26, 32) includes means 
for determining the average amplitude of the video 



signal and indicating an event (Event 2) when the 
average amplitude of the video signal changes by 
a predetermined amount. 

8. The system recited in claim 4 wherein the 
5 event determining means (26, 32) includes means 

for comparing the video envelopes of two frames. 

9. The system recited in claim 8 wherein the 
video envelope comparing means includes means 
for comparing the envelopes of at least one field of 

w two consecutive frames. 

10. The system recited in claim 8 wherein 
video envelope comparing means includes means 
for comparing the envelopes of a least one field of 
two non-consecutive frames. 

75 11. The system recited in claim 10 wherein the 

video envelope comparing means includes means 
for comparing the video envelopes of two frames 
spaced at approximately 20 to 40 frame intervals. 

12. The system recited in claim 4 wherein the 
20 event determining means (26, 32) includes means 

for monitoring the video envelopes of two frames, 
means for obtaining a predetermined number of 
samples from each envelope, means for comparing 
the magnitudes of counterpart samples to obtain a 
26 difference in counterpart samples, and means for 
taking the average of the differences and for in- 
dicating an event (Event 2) whenever the average 
exceeds a predetermined level. 

13. The system recited in claim 12 wherein the 
30 comparing and counterpart samples difference tak- 
ing means includes means for taking the difference 
of the absolute values of the counterpart samples. 

14. The system recited in claim 4 wherin the 
event determining means (26, 32) includes means 

35 for predetermining the average level of the video 
signal, means for determining the amplitudes of 
predetermined protions of the video signal, means 
for comparing the amplitudes of the predetermined 
portions of the video signal with average amplitude 

40 of the video signal and assigning one of a first and 
a second designation to each portion depending on 
whether the amplitude of the predetermined portion 
exceeds or is less than the average amplitude, 
means for utilizing the designations so obtained to 

45 obtain a frame code, and means for comparing the 
frame code with the frame code of a previous 
frame and indicating a change if the current frame 
code differs from the previous frame code by a 
predetermined amount. 

50 15. The system recited in claim 1 wherein the 

signal to be identified inlcudes a video signal and 
said video signal includes a color signal, and 
wherein the feature string extracting means (26, 28, 
32 r 34, 30) includes means (26, 32) for determining 

55 the time interval between predetermined changes 
in the color signal. 
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16. The system recited in claim 15 wherein the 
color signal change determining means (26, 32) 
includes means for comparing a function of the 
color signal with the average of the color signal and 
determining the time intervals when the function of 
the color signal exceeds the average of the color 
signal and the time intervals when the function of 
the color signal is less than the average of the 
color signal, to extract the feature string. 

17. The system recited in claim 16 wherein the 
function of the color signal and the average of the 
color signal is determined for only a single line of 
the frame. 

18. The system recited in claim 17 wherein the 
single line is near the top of the image. 

19. The system recited in anyone of the pre- 
ceding claims wherein the signals to be identified 
are representative of program material, character- 
ized in that said predetermined changes are sec- 
ond predetermined events (Event 2) as contrasted 
to first predetermined events (Event 1) relating to 
the using mode on the part of the program con- 
sumers. 

"20. The system recited in claim 19 for identify- 
ing programs viewed on a television receiver (12), 
wherein the programs are obtained from a video 
recorder (14) and from an external program source 
(16, 18), characterized by 

- means responsive to predetermined operating 
conditions of said television receiver (12) and said 
video recorder (14), 

- logic means responsive to the operating con- 
ditions responsive means for providing an indica- 
tion if the program being viewed is obtained from 
the video recorder (14) or from the external pro- 
gram source (16, 18), and 

- means for identifying the program being viewed, 
said identifying means being reponsive to said 
logic means and operative to identify said program 
according to two different criteria depending on 
whether the program being viewed is obtained from 
the external program. source (16, 18) or from the 
video recorder (1 4). 

21. The system recited in claim 20 wherein 
said operating conditions responsive means in- 
cludes means responsive to the on/off condition of 
the television receiver (12) and means responsive 
to the on/off condition and the piay/record condition 
of the video recorder (14). 

22. The system recited in claim 20 or 21 
wherein said logic means includes means (26) for 
detecting during viewing said first predetermined 
events (Event 1) relating to the viewing mode, and 
wherein said program identifying means includes 
means (26) responsive to the detection of said first 
predetermined event for detecting during viewing a 
second predetermined event (Event 2) in the pro- 
gram material. 



23. The system recited in claim 22 wherein 
said program identifying means further includes 
means (28) responsive to the detection of said 
second predetermined event (Event 2) for extract- 

5 ing during viewing at least one predetermined char- 
acteristic of the program material. 

24. The system recited in claim 23. for deter- 
mining the viewing habits of at least one household 
and having a respective number of home units 

w (10) and a central unit (40), characterized in that 
said operative conditions responsive means and 
said logic means exist at each home unit (10) and 
the home unit further includes means (30) for stor- 
ing the predetermined characateristic of the viewed 

75 program material and the time that the predeter- 
mined characteristic is extracted. 

25. The system recited in claim 24, further 
including - means (26) for continuously detecting 
during recording second predetermined events 

20 (Event 2) in the program material, 

- means (28) responsive to the detection of said 
second predetermined events for extracting during 
recording at least one predetermined characteristic 
of the program material, and 

25 - means (30) for storing during recording the pre- 
determined characteristics and the times that the 
predetermined characteristics are extracted. 

26. The system recited in claims 24 and 25, 
including means (48) for comparing the predeter- 

30 mined characteristics stored during viewing and 
recording and their times of occurrence with refer- 
ence predetermined characteristics stored in the 
central unit (40) that occurred at substantially the 
same time to identify the programs viewed and 

35 recorded. 

27. The system recited in anyone of the claims 
24 to 26, further including 

- means (26) for detecting during playback the 
second predetermined events (Event 2) in the pro- 

40 . gram material, 

- means (28) responsive to the detection of said 
second predetermined events (Event 2) for exac- 
ting during playback at least one predetermined 
characteristic of the program material, 

45 - means (30) for recording during playback the 
predetermined characteristic, and 

- means (48) for comparing the predetermined 
characteristic recorded during playback with refer- 
ence predetermined characteristics stored at the 

so central unit (40) at the time of recording to identify 
the program. 

28. The system of claim 27 wherein said sec- 
ond predetermined events detecting means (26) 
operative during playback includes means for de- 

55 tecting during playback the respective first pre- 
determined event (Event 1), and means (26) re- 
sponsive to the detection of said first predeter- 
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mined event for detecting during playback the sec- 
ond pedetermined events (Event 2) in the program 
material. 

29. The system of claim 27 or 28 including 
means (26, 28, 32, 34) at the home unit (10) for 5 
comparing the time intervals between the detected 
events during playback and the time intervals be- 
tween the events stored during recording, said pre- 
determined characteristics comparing means (48) 
being responsive to a match provided by the time 10 
intervals comparing means (26, 28, 32, 34). 
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